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ABSTRACT 

This report summarizes major findings and policy 
implications of a University of Pittsburgh project titled 
Methodological Research on Knowledge Use and School Impxovement. The 
major emphasis has been methodological rather than substantive. The 
primary purpose has been to describe, evaluate, and recommend 
alternative concepts, methods, and techniques that enlarge capacities 
to understand and shape processes of knowledge use and school 
improvement. This project focuses on three core methodological 
problems: the criterion problem in knowledge use, the multiattr ibute 
problem of meanings attached to knowledge, and conceptualization and 
measurement in the knowledge transactional problem. Procedural 
inventory, typological procedures and analytic procedures are 
discussed as research objectives. The approach employed to realize 
these objectives may be described as a form of methodological 
triangulation : documentary and archival search; pooling, selection, 
and analysis of procedures; and network consultation, conferences, 
and workshops. The project results are summarized around five sets of 
methodological findings: reproducibility, reliability, and validity; 
knowledge use as a criterion; formal classification schema; 
generative classification schema; and sociocognitive grid procedures. 
Appendices contain an annotated bibliography of working and 
occasional papers, a sample procedure abstract, a sample study 
profile, and frequency distributions for study profiles. 
(Author/PN) 
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Section 1 



INTRODUCTION 

This report summarizes major findings and policy implications of a pro- 
ject titled Methodological Research on Knowledge Use and School Improvement. 
The major emphasis of this twelve-month project, formally initiated in 
January 1981, has been methodological rather than substantive. Our primary 
purpose, therefore, has been to describe, evaluate, and recommend alternative 
concepts, methods, and techniques that enlarge capacities to understand and 
shape processes of knowledge use and school improvement. We have been concerned 
only indirectly with the production of new substantive research findings. 

Methodological research, in the sense we understand this term, is about 
some collection or set of methods and techniques; it is not itself one of 
the set.^ Methodological research therefore involves the systematic invest- 
igation of alternative standards--logical , empirical, practical—for choos- 
ing particular research procedures. Many of the most important questions 
surrounding the study of knowledge use are, in this sense, methodological: 

*What do we mean by knowledge when we attempt to measure its 
uses by parents, teachers, principals, and other stakeholders 
in school improvement? 

*How might we classify and measure the uses of work-related 
knowledge and hbw are thses uses related, if at all, to 
characteristics of knowledge itself? 



See Paul Watzlawick, John Weakland, and Richard Fisch, Change : Principles 
of Problem Formation and Problem Resolution (New York: N.W. Norton, 1974, 
p. 6. Compare Whitehead and Russell: "Whatever involves all of a collection 
must not be one of the ool lection." Principia MathematicaTTsTO) , p. 101. 
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*What procedures are available to develop more appropriate 
constructs of knowledge use, given the presence of diverse and 
frequently conflicting conceptions of knowledge and its uses 
for school improvement? 

*How might we classify and measure criteria employed by educators 
to assess the adequacy, relevance, and cogency of knowledge 
introduced into school settings? 

*If school improvement is a collective process, involving a 
system of interdependent relations among principals, parents, 
teachers, and other educators, how might we investigate the 
impact of knowledge use on collective as well as individual 
actions? 



Section 2 
CORE METHODOLOGICAL PROBLEMS 

Questions posed above suggest a range of pressing methodological 
problems confronting those who study, and thereby seek to improve, processes 
of knowledge use and school improvement. The present project has focused 
on three such methodological problems, problems that represent a core of 
unresolved needs and opportunities facing researchers and practitioners 
in the field. 

2.1 The Criterion Problem 

What is knowledge use? Answers to this basic question have assumed a 

variety of forms, none of which has yielded a satisfactory account of the 

criteria in terms of which we may recognize the use of knowledge when 

2 

actually it occurs. Many scientists, policymakers, and practitioners 
continue to visualize knowledge, information, or research as quasi -physical 



Systematic reviews of this question include Judith K. Larsen, "Knowledge 
Utilization: Current Issues," in The Knowledge Cycle , ed. Robert F, Rich 
(Beverly Hills, CA: Sage Publications, 1981); Fritz Machlup, Knowl edge 
and Knowledge Production , Vol. 1 (Princeton, NJ: Princeton University Press, 
T580); and Carol H. Weiss, "Measuring the Use of Evaluation," in Utilizing 
Evaluation: Concepts and Measurement Techniques , ed, James A. Ciarlo (Beverly 
Hi n s , CA: Sage Publ i cati ons , 1981). See also Volume 2 of this report, 
especially papers by Gerald Zaltman, "Construing Knowledge Use;" and William 
N. Dunn, "Methodological Choices in Studying Knowledge Use." Contents of 
^ these and other papers in Volume II are summarized in Appendix A below- 
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entities embodied in externally visible objects. Prevalent metaphors are 
3 

illustrative: 

♦Knowledge is carried from source to receiver like a bucket 
carries water 

♦Knowledge is transported from producer to user like a 
wheelbarrow carries sand 

♦Knowledge is delivered from producer to user like a lumber- 
jack fells trees for mills down-river 

♦Knowledge travels like a bullet shot at a target 

♦Knowledge is introduced like a hypodermic needle injects 
vaccine 

♦Knowledge spreads like an infectious disease 

♦Knowledge is stored like grain in a silo, books in a library, 
or bits, bytes, or blocks in a computer 

♦Knowledge is fabricated like the production of outputs from 
inputs in a technological or machine process 

Physical mv^taphors such as these justifiably may be regarded as parts 
of the deep structure^ of present-day thinking about knowledge use, generally 
and in settings of educational practice. Although physical metaphors re- 
present one aspect of the domain we seek to understand (it is difficult to 
imagine knowledge without libraries, research reports, or information systems), 
they nevertheless constrain or distort our understanding of processes of 
knowledge use. Physical metaphors, for example, may preempt opportunities 
to visualize knowledge use as a cognitive process that is at once central 



Compare Everett M. Rogers and D. Lawrence Kincaid, Conwnuni cation Networks: 
Toward a New Paradigm for Research (New York: The free Press, 1981), pp. 
38-9; and Ernesc R. House, "Three Perspectives on Innovation," in Improving 
Schools: Using What We Know , ed. Rolf Lehming and Michael Kane (Beverly 
Hills, CA: Sage Publications, 1981). 

4 

See House, "Three Perspectives on Innovation and Donald Schon, 

"Generative Metaphor: A Perspective on Problem Setting in Social Policy," 
in Metaphor in Thought , ed. A. Ortony (Cambridge: Cambridge University 
Press, 1979). 



to school improvement but entirely inaccessible to direct inspection. Thus, 
to employ a contrasting metaphor, knowledge use is similar to drawing a 
mental map: 

Everyone employs a "cognitive map," that is, to define it meta- 
phorically, a picture in their mind of the complex phenomena with 
which they deal. One's map of instruction may be very simple- 
certain things to be learned, pupils to learn them, and learning 
activities through which the pupil will learn the desired things. 
But teachers quickly discover that this map is too simple to help 
them identify what is wrong when the pupils do not learn even 
though they go through the activities. They then look for refine- 
ments in their maps, additional concepts that help them to under- 
stand the phenomena they encounter as teachers, and they will add 
and modify their maps from time to time as they become aware of 
new concepts that appear to give greater richness to the represent- 
tation they have of the situation. 

The clash of contemporary metaphors highlights the complexity of 
knowledge use as a phenomenon-to-be-understood. Knowledge use cannot 
be satisfactorily defined as a physical or quasi -physical process; nor 
is the meaning of the term adequately captured by such terms as "social," 
"political," "cultural," or even "cognitive . Knowledge use is all of 
these and more, and it is precisely this multidimensional ity that creates 
the problem. At present we lack an adequate understanding of what is 
meant by the term knowledge use, even though knowledge, information, or 
research use are frequently taken as criteria of success (and failure) 
by policy-makers, practitioners, and scientists. Under these circumstances 
it becomes more than a little doubtful if members of the research and 
policy-making communities are even equipped to study or shape processes 
of knowledge use. 



The lack of accessibility to direct inspection does not mean that 
cognitive processes cannot be classified, measured, or inferred; nor does 
measurement oer se imolv the corruption of subjective experiences. See 

Volume III of this report: Measuring Knowledge Use: A Procedural 
Inventory. 

^Ralph W. Tyler, "How Schools Utilize Educational Research and Development," 
in Research and Development and School Change , ed. Robert Glaser (Hillsdale, 
N.JTI Lawrence Erlbaum Associates, 1978), p. 96, 
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2.2 The MuUi attribute Problem 

Why does knowledge vary in adequacy, relevance, and cogency? This 
questions calls attention to the fundamentally interpretive character of 
processes of knowledge use, processes in which different stakeholders 
attribute multiple and frequently conflicting properties to knowledge and 
its uses. The meanings attached to knowledge and its uses typically 
arise from diverse sets of assumptions held by different stakeholders in 
school improvement. These diverse sets of assumptions are themselves 
organized in terms of what we and others have called frames of reference, 
a concept with a long if ambiguous history in the social and natural 
sciences.^ 

The concept frame of reference performs an essential function 
in theory and research on knowledge use. In its widest sense the term 
refers to the orientational framework to which particular physical, social, 
and cultural objects are related, providing them with contextual as well 
as referential meanings. Social and cultural objects such as those found 
in settings of educational practice— for example, technical assistance 
roles, computer-assisted instruction, standardized achievement tests-- 
derive their meanings from contexts, as well as from external referents for 
these objects. These contexts are established by frames of reference. 

Frames of reference are the equivalent of what others have called 
construction systems--organized sets of criteria, standards of assessment, 
or assumptions which facilitate and restrict a person's range of cognitive 

g 

and social activities. Thus, processes of knowledge use are structured 

^See Burkart Holzner and Leslie Salmon-Cox, ''Frames cf Reference and 
the Prediction of Knowledge Use,*' in Volume IT of this report. 

o 

See George A. Kelly, The Psychology of Personal Constructs , Vols. 1 and 2 
(New York: N.W. Norton, 19557^ 
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by the ways that educational stakeholders anticipate or predict events, 
such anticipation being a function both of individual and collective 
frames of reference and of the social frameworks in which reference frames 
are established, maintained, and changed.^ 

Frames of reference, in a more restricted and operational sense, in- 
clude what we and others have called truth, utility, and cogency tests. 
Truth tests are decision rules or standards for assessing the adequacy 
of knowledge claims, while utility or relevance tests delineate appropriate 
domains of inquiry or action. Cogency tests, by contrast, establish the 
relative force or confidence required of knowledge in particular circum- 
stances. Tests of truth, relevance, and cogency represent diverse and 
frequently conflicting standards of assessment brought to bear in appraising 
and acting on educational innovations. 

Until recently it has been unclear whether concepts of truth, relevance, 
and cogency exhaust the classes of standards employed to assess knowledge 
claims. Hence, each of these broad classes of tests might be divided into 
additional categories: pragmatic, ethical, authenticative, causal, and 
so forth. Needed are procedures for generating a comprehensive class- 
ificational schema or typology that not only differentiates broad classes 
of standards for assessing knowledge but also specifies particular tests 
employed by different stakeholders in school improvement. Such procedures 
would facilitate enormously current efforts to map the universe of meaning 

See Holzner and Salmon-Cox, "Frames of Reference and The Prediction of 
Knowledge Use," Volume II. 

^^See Holzner and John Marx, Knowledge Application : The Knowledge System 
in Society (Baston: Allyn and Bacon, 1979); Carol H. Weiss and MichaiT 
TT Bucuvalas, "Truth Tests and Utility Tests: Decision Makers' Frames of 
Reference for Social Science," The American Sociological Review, 45, 2(1980): 
302-12; and William N. Dunn, "Reforms As Arguments," Volume II and Knowledge , 
3, 3(1982): 293-326. See also Donald T. Campbell, "Experiments As Arguments, 
Knowledge , 3, 3(1982) :327-338. 

Q ^^Dunn, "Reforms As Arguijents," Volume II. 
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surrounding knowledge use and school improvement. 
2.3 The Transactional Problem 

How can we conceptualize and measure knowledge transactions? A com- 
prehensive classification of truth, relevance, and cogency tests, while 
important for mapping the universe of meaning surrounding school improve- 
ment in general, would not deal with the distribution of these tests among 
individual and collective actors; nor would such a classificatonal schema 
identify unique or distinctive structural properties of particular frames 
of reference. Multiple and potentially conflicting truth and relevance 
tests may be applied by the same person, while the structure of individual 
as well as collective frames of reference may change over time as a result 
of the dynamics of social relations among these who affect and, in turn, 
are affected by school improvement. 

What in now unclear about processes of knowledge use, generally and 
in educational settings, is the extent to which changes in frames of 
reference affect and, in turn, are affected by patterned social relations.^ 
Indeed, it is possible and desirable to visualize this problem as analogous 
to a single coin with two porous and semi -permeable surfaces, the '^social" 
and the "cognitive". Processes of knowledge use, according to this form- 
ulation, may be viewed as sociocognitive relations among actors engaged in 



The term patterned social relations is deliberately used to avoid certain 
well-established but prematurely specified constructs employed to label, 
categorize, and generalize social processes, including "stages of innovation," 
"linker", "loosely coupled system," "knowledge transfer," and so forth. 
The empirical validity of these constructs, and the causal models in 
which they are often embedded, is at present doubtful if lacking. On 
these points see Matthew B. Miles, "Mapping the Common Properties of 
Schools"; and Paul Berman, "Educational Change: Toward An Implementation 
Paradigm", in Lehming and Kane, op . cit . 
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generative reciprocal acts, that is, acts involving cognitively structured 
(filtered, mediated, processed, interpreted, construed) feedback that 
transforms and is transformed by patterned social relations. The essential 
property of such sociocognitivo relations is that they involve subject- 
subject and subject-object relations that cannot be decomposed or reduced 
to erstwhile independent dimensions conveniently labelled as "social" and 
"cognitive," "instrumental" and "conceptual," or "behavioral" and "attitu- 
dinal."^^ 

A recognition of the contextual, relational, and generative properties 

of knowledge use has promoted significant shifts in the language of research. 

For example, the terms interaction, exchange, and transfer have been 

replaced with that of transaction,^^ while the term receiver has been dis- 

IS 

carded in favor of transreceiver. Yet existing methodologies for studying 
knowledge use do not ad&qii_.:,2iy address generative reciprocal processes 
involving cognitive as well as social processes. While recent contributions 
to theory and research on communications networks recognize the importance 
of distinguishing contextual and referential meanings, there have been 
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Compare Buckley: *'person-to-person orientations are virtually never made 
and sustained in an environmental vacuum. . .person-to-object orientations 
are seldom, if ever, made in a social vacuum..." Walter Buckley, Sociology 
and Modern Systems Theory (Englewood Cliffs, NJ: Prentice-Hall, 1967), p. 114. 

14 

See, for example, Raymond A. Bauer, "The Obstinate Audience: The Influence 
Process form the Point of View of Social Communication,'* American Psychologist, 
19(1964):319-28; Gerald Zaltman, "Knowledge Utilization As Planned Social 
Change," Knowledge , 1, 1(1979) :82-105; and Dunn, "Reforms As Arguments," 
Volume II. 

15 

Ithiel de Sola Pool, "Communication Systems," in Handbook of Comrrtunication , 
ed. I. de Sola Pool and W. Schramm (Chicago: Rand McNally, itTsT. 

^^William N. Dunn, Ralph Bangs, and Hassan Rahmanian, "Stjdying Knowledge 
Networks," Volume 2. 

^^Everett M. Rogers and D. Lawrence Kincaid, Communications Networks : Toward 
a New Pardigm for Research (New York: The Free Press, 1981), p. 1-78. 

1 ^ 
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few and scattered attempts to develop systematic and reproducible procedures 
for identifying organized configurations of assumptions, constructs, or 
criteria applied by members of different sociocognitive networks. While 
research methods used to study the social and cognitive (intellectual) 
properties of scientific networks approach what we have in mind by 
systematic and reproducible procedures, the sociocognitive organization 
of scientific communities appears to be much less complex than those practice 
settings with which knowledge use researchers are typically concerned. 
The problem of knowledge transactions, then, is one of determining how 
best to conceptualize and measure complex, generative reciprccal processes 
that are at once cognitive and social. 

Section 3 
RESEARCH OBJECTIVES 

Core methodological problems discussed in the preceding section may 

be specified in many ways. We chose at the outset of the project to focus 

20 

on three major objectives; 
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See, for example, Diana Crane, Invisible Colleges (Chicago: University of 
Chicago Press, 1972); Nicholas C. Mull ins. Theories and Theory Groups in 
Contemporary American Sociology (New York: Harper and Row, 1973); and 
Harriet Zuckerman, Scientific Elite; Nobel Laureates in the United States 
(New York: The Free Press, 197Tr 

19 

Compare, for example, the sociocognitive complexity of schools reported 
by A. Michael Hubennan, "Recipes for Busy Kitchens: A Situational Analysis 
of Routine Knowledge Use in Schools," Knowledge (forthcoming). We join Rogers 
and Kincaid (op.cit. , p. 32) in endorsing a .statement attributed to H. Russell 
Bernard and others: "As far as we are aware, content of communication is a 
topic universally avoided by researchers in social network theory." We would 
add that this statement applies with equal force to research on knowledge use 
in general. 

20 

See William N. Dunn and Burkart Holzner "Knowledge Use and School Improvement; 
Conceptual Framework and Study Design," Volume II. 



♦Development of user-responsive research instruments 

♦Construction of a grounded typology of criteria used 
to appraise knowledge 

♦Development of decision-focused analytic procedures for 
investigating complex knowledge appraisals 

During the course of the project these objectives were enlarged, elaborated, 
and redefined as a consequence of our own learning experiences. 

3.1 Procedural Inventory 

The aim of developing user-responsive research instruments reflected a 
cotmiitment to research procedures that are likely to elicit the knowledge- 
in-use of stakeholders in school improvement.^^ Early in the project it 
became evident that the development, pilot-testing, and provisional valid- 
ation of such instruments presupposed a systematic review and evaluation of 
instruments available to knowledge use researchers, generally and in the 
field of education. Research on knowledge use, it also became clear, has 

been conducted by many investigators who have not always (or often) labelled 

22 

their activities in this way. Finally, the original focus on research 
instruments —that is, more or less carefully calibrated procedures designed 
to measure properties whose meanings have been defined in advance by 
researchers— soon gave way to a broader concern with research procedures 
in general . 

Thus, the first of our original project objectives was redefined and 



On the concept of knowledge-in-use see Burkart Holzner and Evelyn M. 
Fisher, "Knowledge-In-Use: Considerations in the Sociology of Knowledge 
Applications," Knowledge , 1, 2(1979) :219-44. 

22 

For example, knowledge use has seldom been a common term among investigators 
in various disciplinary specialties (e.g., cognitive psychology, cognitive 
sociology, ethnography) and interdisciplinary fields (e.g., planned social 
change, social marketing, decision theory, comnuni cations research). 
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enlarged to include an inventory of procedures available for the study of 
23 

knowledge use. The redefinition of this objective was particularly impor- 
tant in certain areas— for example, the broad domain of qualitative method- 
ology—where the term research instrument is often inappropriate.^^ The 
enlargment of this objective, since it also required an investigation of 
procedures employed beyond the boundaries of knowledge use research proper, 
likewise made it unfeasible to develop, pilot-test, and validate more than 
a few new or promising research procedures. We therefore relied on 
reliability and validity information supplied by developers of procedures. 

3.2 Typological Procedures 

The aim of constructing a grounded typology of criteria used to 
appraise knowledge was based on a recognition that the concept of frame of 
reference, together with its specification in the form of truth and utility 
tests, has not been properly validated in settings of practice. In the 
initial phase of the project investigators reviewed, critiqued, and modified 

an original typology of tests employed to assess knowledge adequacy, 

26 

relevancy, and cogency. At the same time it b^-ame increasingly clear 
that a grounded typology— that is, a typology whose categories are coordinated 



23 

See Volume III below. 



24 

See Thomas R. Mclntyre and Evelyn M. Fisher, "Qualitative Procedures for 
Research on Knowledge Use and School Improvement." Volume II. 

25 

Several doctoral dissertations and one university- funded research project 
have incorporated procedures identified or developed in this project. These 
projects, now at various stages of completion, focus on processes of know- 
ledge use in education, mental health, local government, criminal justice, 
international relations, and state-local program evaluation. 

26 

Dunn, "Reforms As Arguments," and Holzner and Salmon-Cox, "Frames of 
References...," both in Volume II of this report. 

O 1 
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with the subjectively meaningful constructs of principals, parents, teachers 
and other educators— would require substantial field research of a type 
outlined in a proposal submitted to NIE midterm in the present project. 
We also concluded that a field study based on available qualitative methods 
would be unlikely to contribute significantly to the construction of a 
grounded typology. Although many such studies have been conducted in the 
past, our assessment of methods used to conduct these studies persuaded 
us that new procedures were necessary. Existing procedures, while they do 
elicit subjectively meaningful experiences of educational stakeholders, can 
seldom be reproduced by other investigators or examined as part of an 
external audit that attempts to ensure that methodological accountability 
of research findings. 

The second of our original project objectives was therefore redefined 
to emphasize the identification and development of procedures for generating 
grounded typologies of knowledge-in-use. While we had followed and appreciated 

related attempts to conceptualize, measure, and type decision makers' frames 
29 

of reference, we nevertheless adopted an altogether different methodological 



27 ■ 

Transacting Knowledge for Local School Improvement (Pittsburgh, PA- 
University of Pittsburgh, Program for the Study of Knowledge Use, May 1981) 

28 

See William N. Dunn, "Methodological Choices in Studying Knowledge Use," 
Volume II. Compare A. Michael Huberman and Matthew B. Miles, "Drawing 
Valid Meaning from Qualitative Data: Some Techniques of Data Reduction 
and Display," Paper prepared for a Symposium on Advances in the Analysis 
of Qualitative Data, Annual Meeting, American Educational Research 
Association, New York, NY, March 1982. The term "external audit" is used 
by Egon G. Guba in "Criteria for Assessing the Trustworthiness of 
Naturalistic Inquiries," Educational Communication and Technology 
Journal (Spring 1981). 

29 

Especially Carol H. Weiss with Michael J. Bucuvalas, Social Science 
Research and Decision Making (New York: Columbia University Press, 1981). 
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course. Accordingly, we avoided procedures designed to measure assumedly 
universal (nomothetic) constructs, traits, or factors, attempting instead 
to identify and develop procedures for generating contextually specific 
(idiographic) constructs that are subjectively meaningful to stakeholders 
in school improvement. We also insisted that any such generative 
procedure satisfy the requirement of reproducibility, since it is the 
failure to meet this standard that has fatally flawed so many otherwise 
useful'.qualitative investigations. 

3.3 Analytic Procedures 

The aim of developing decision-focused analytic procedures was 
motivated by the awareness that processes of knowledge appraisal are 
complex. This complexity is partly a consequence of diverse social 
relations among stakeholders who shape collective decisions;^^ it is also 
a function of diverse cognitive relations within and among stakeholders 
with distinctive frames of reference. Thus, we did not limit decisions 



30 " 

By reproducibility we mean that a procedure must be sufficiently orderly, 
regular, and specific to permit repeated applications by successive invest- 
gators. Replication, by contrast, refers to the quality by virtue of which 
studies and/or research findings may be repeated. Replicable studies (e.g. 
laboratory experiments) require reproducible procedures, but the use of 
strategies of qualitative research (e.g., multiple triangulation) . The 
issue of reproducibility (not replication) is centra-1 to recent critiques 
of qualitative methods, for example, Matthew B. Miles, "Qualitative Data 
as an Attractive Nuisance: The Problem of Analysis," Administr ative 
Science Quarterly . 24(1979) : 590-601; and Huberman and Miles, op. cit. 

31 

See Evelyn M. Fisher, "Contexts for Conducting Field Research on Know- 
ledge Use and School Improvement," Volume II. 

32 

See Holzner and Salmon-Cox, "Frames of Reference and The Prediction of 
Knowledge Use;" and Dunn, "Usable Knowledge: A Metatheory of Policy Research 
in the Social Sciences," Volume II. 
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to those involving action, but included as well various forms of subjective 
judgment and appraisal. 

The early emphasis on decision-focused analytic procedures was prompted 
by an awareness of opportunities to explore the applicability of various 
forms of multiattribute decision theory. These procedures not only 
permit the modeling of judgment processes, but provide various heuristics 
designed to improve individual and collective decisions. The limitation of 
these procedures, however, is that none is based on systematically acquired 
information about frames of reference. For this reason it is not possible 
to make inferences about the effects of differences in frames of references 
(e.g., the dominance of a particular kind of truth test) on individual and 
collective judgments. 

Thus, the third of our original objectives was enlarged to include the 
identification and development of procedures that would permit: 

*The analysis and measurement of structural properties of frames 
of reference, including differentiation, integration, complexity, 
permeability, and substantive organization; 

*The analysis and measurement of relations among these struc- 
tural properties in contexts of social decision making and 
collective action; and 

*The joint analysis and measurement of social and cognitive 
relations as these change over time. 

The enlargment, elaboration, and redefinition of this and other original 
project objectives contributed to a more ambitious but focused research 
effort. 



These include multiattribute utility analysis, analytic hierarchy analysis, 
social judgment analysis, strategic assumption surfacing and testing, 
conjoint analysis, and functional measurement. See William N. Dunn and Mary 
Jo Dukes, "Multiattribute Procedures for Adoption Decision Making,'* Volume II. 
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Section 4 
APPROACH 

The approach employed to realize these objectives may be described 
as a form of methodological triangulation.^^ Thus, we attempted through- 
out the project to explore, anticipate, or document the consequences- 
theoretical, conceptual, technical, empirical, practical --of employing 
alternative methods to investigate processes of knowledge use and school 
improvement. 

4.1 Documentary and Archival Search 

An important project activity was to conduct a search of relevant 

books, articles, reports, and papers on knowledge use and school improve- 

35 

ment. A select annotated bibliography was compiled on the basis of five 
criteria: insight into processes of knowledge use; factors impeding know- 
ledge use; strategies for improving knowledge use; elaboration of theories 
of knowledge use; and most importantly, attempts to define knowledge use 
as a criterion dimension (or "dependent variable"). Annotated items that 
satisfied one or several of these criteria were identified through a 
six-stage process that ranged from consultation with content experts in 
fields of education and knowledge use to the use of indexes (Social Science 



The focus on methodological as distinguished from substantive issues meant 
that it was rarely possible to employ observer and data triangulation. For 
this reason the approach may be viewed as! a truncated version of multiple 
triangulation as described, for example, by Norman K. Denzin, The Research 
Act (Chicago: Aldine, 1970). 

"^^Mary Jo Dukes, "Knowledge Use and School Improvement: A Select Annotated 
Bibiligraphy," Volume II. 
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Citation Index, Social Science Periodical Index, Current Index to Journals 
in Education), specialized bibliographies, handbooks, and an archive of 
measures of creativity and innovation at Northwestern University. 

One of the major aims of the documentary and archival search was to 
identify new or promising procedures for conceptualizing and measuring 
knowledge use. Archival materials often included detailed descriptions of 
procedures while, in some cases, available handbooks contained summaries 
of procedures that were judged to be relevant to the definition and 
measurement of knowledge use. Characteristically, most books and nearly all 
articles and reports failed to provide an adequate description of procedures 
employed to conduct research. For this reason it was necessarily to 
contact investigators directly. 

The documentary and archival search yielded a list of investigators 

who had reported results obtained form different research procedures: 

questionnaires, interview schedules, content analysis, case analysis, and so 
37 

forth. This list was enlarged by adding members of our national 
advisory network and other networks of researchers and practitioners 
associated with the National Institute of Mental Health, the National 
Science Foundation, and National Institute of Education. Also included 
in this list were persons who had attended a 1981 Conference on Knowledge 
Use organized by our program. Over one-hundred investigators were contacted 
by letter and/or phone and asked to provide a description of the procedures 
they had employed to conduct their research project(s), including copies 



For example. Dale G, Lake, Matthew B, Miles, Robert B. Earle (eds,) 
Measuring Human Behavior (New York: Columbia University Press, 1973) 

37 

See Dunn et al., ''Measuring Knowledge Use: A Procedural Inventory", loc. cvt, 
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of interview protocols, self-administered questionnaires, tests, or coding 
schema. 

4.3 Pooling, Selection, and Analysis of Procedures 

The documentary and archival search, together with inquiries to invest- 
igators, yielded a pool of some two-hundred candidate procedures. Our next 

task was to select from this pool procedures with maximum relevance to 

38 

methodological research problems addressed in the project. The original 

set of candidate procedures was eventually narrowed to some sixty-five 

procedures that were compared, co'^^rasted, and evaluated in accordance with 

a standardized abstracting procedure. Each abstract provides a general 

39 

information profile including author, availability, purpose, variables, 
description, development, reliability/validity and administration. In 
addition, we wished to investigate the study designs in terms of which 
procedures were employed. For this purpose we constructed, where possible, 
a study profile^^ describing 'the unit of analysis, sampling procedure, 
type of design, research methods, practice area or population, definition 
of knowledge use, and other key methodological characteristics of studies. 

Study profiles were limited to major studies of knowledge use identified 
iR the course of the project, since it would be unmanageable to examine 
characteristics of study designs which yielded new or promising procedures 
(e.g., cognitive mapping techniques) but which were not expressly directed 
at the conceptualization and measurement of knowledge use. General inform- 
ation profiles were limited to procedures that satisfied two criteria: 
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.^^Namely, the criterion problem, the multiattribute problem, and the 
lljjansactional problem (see Section 2 above). 

^^See Appendix B of this report. 
An 

,^ See Appendix C of this report. 



reproducibility and face relevance to knowledge use. The criterion of 
reproducibility required that a given procedure be sufficiently specific, 
regular, and orderly that its steps may be repeated by some other investigator 
By applying this criterion of inclusion we were forced to abandon a large 
number of candidate procedures; many procedures employed by investigators 
in the field do not meet this standard. In some instances--especially 
those involving qualitative methods—researchers have explicitly challenged 
or disavowed the appropriateness of reproducibility as a criterion. 

The second criterion— face relevance to the study of knowledge use-- 
required judgments about the conceptualization of knowledge and its uses, i 
Here we were guided by general considerations that procedures should permit 
the acquisition of information about multiple dimensions of knowledge use; 
that procedures should be appropriate for investigating alternative theories; 
and that procedures should somehow elicit information about cognitive 
properties, since it is this feature that provides the field with ^ unique 
purpose and rationale. The criterion of face relevance, when applied to 
research on knowledge use proper, resulted in the inclusion of some forty-two 
procedures. An additional 22 procedures were included as relevant, 
even though they had not been developed or used by researchers who view 
their work as falling within the domain of knowledge use per se. 

4.4 Network Consultation, Conferences and Workshops 

Throughout the project investigators were in frequent contact with 
various members of a national advisory network of scholars and practi- 
tioners in the field of knowledge Use. Portions of working papers and 
occasional^Japers were reviewed by members of this network or presented 



See Appendix A. 
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at a variety of conferences and workshops: 

* National Conference on Knowledge Use, University 
of Pittsburgh, Program for the Study of Knowledge 
Use, March 1981 

* Annual Meeting, American Educational Research Asso- 
ciation, Dissemination and Utilization Researchers 
Group, Los Angeles, March 1981 

* Conference on the Utilization of Knowledge Acquired 
Abroad, University of Pittsburgh, Center for Inter- 
national Studies, October 1981 

* Workshop on the Utilization of Educational 
Technologies, Pittsburgh, Metropolitan Pittsburgh 
Broadcasting Company, September 1981 

* Workshop on Long-Range Planning for Improvement, 
Pennsylvania Department of Education and University 
of Pittsburgh School of Education, June 1981 

* Luncheon Workshops on Knowledge Use, University of 
Pittsburgh, Program for the Study of Knowledge Use, 
January 1981-June 1982 

* Network of Consultants on Knowledge Transfer, the RAND 
Corporation and the National Institute of Mental 
Health, May 1981 

* Annual Meeting, American Educational Research Asso- 
ciation, Dissemination and Utilization Researchers Group, 
New York, March 1982 

* Conference on Knowledge Use: Theory and Methodology, 
Communications Institute, East-West Center, 
University of Hawaii, April 1982 

Theso activities permitted continuous interaction and the sharing of 
ideas and materials with numerous scholars and practitioners in the 
field, thus enlarging our capacity to achieve project objectives. 



ERLC 



20 



Section 5 
SUMMARY OF METHODOLOGICAL FINDINGS 

A documentary record of research carried out under this project is 

contained in the working and occasional papers annotated in Appendix A 

and organized as Volume II (Conceptualizing Knowledge-In-Use) and Volume 

III (Measuring Knov/ledge-In-Use) of this report. Although each of these 

papers presents conclusions drawn from the analysis of distinct problems, 

the results of the project as a whole may be summarized around five 

major sets of methodological findings. 

5.1 Reproducibility, Reliability, and Validity 

A striking feature of published and unpublished literature on pro- 
cesses of knowledge use, generally and in the domain of education, is 
the relative paucity of reproducible research procedures. Out of the 
many hundreds of studies surveyed in this project, few are based on 

procedures that are sufficiently orderly, regular, and specific that 

42 

they may be repeated by others. Even where procedures are reproducible 
there are relatively few. -instances where the same procedure has been 
applied by two or more investigators.^^ Thus, knowledge use research appears 
to have the same segmented and non-cumulative character as research carried 
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These procedures have been abstracted in Dunn, Dukes, and Cahill, Volume 
III: Measuring Knowledge-in-Use: A Procedural Inventory. 

43 

Exceptional instances involving multiple applications of the same pro- 
cedure are documented in Volume III. Gene Hall, for example, at the 
Research and Development Center for Teacher Education at the University 
of Texas at Austin, has developed multiple procedures for assessing various 
aspects of the innovation process which have been used repeatedly by members 
of that research team. 
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out in established disciplines. While research on knowledge use is 
a much less mature enterprise that spans many distinct but complementary 
research traditions, it is ironic that investigators in a field coirmitted 
to knowledge-based innovations are themselves not obviously innovative. 

Many individual researchers who have developed reproducible proce- 
dures report reliability data. More than forty percent of procedures for 
which reliability data are appropriate are accompanied by such information. 
While considerably less than half of testing and content-analytic procedures 
are accompanied by test-retest or internal consistency reliability 
coefficients, approximately fifty percent of questionnaires and interview 
schedules make such reliability data available. Overall, only eighteen 
of fifty-eight studies attempted to establish the validity of constructs 
offered to represent processes of knowledge use, even though construct 
validation procedures are specifically designed to examine the status 
of mental constructs of all kinds. This methodological gap is particularly 

significant, given that cognitive constructs are central to the study of 
46 

knowledge use. 



Compare, for example, Bonjean, Hill, and McLemore's finding that little 
more than two percent of some 2,000 scales and indices were used more than 
five times by sociologists. See Charles M. Bonjean, Richard J. Hill, and 
S. Dale McLemore, Sociological Measurement : An Inventory of Scales and 
Indices (San Francisco, CA: Chandler, 1967), p. 9. 

45 

See Volume III, Appendix C. In almost all instances it was impossible to 
determine conclusively whether reliability data were neglected by invest- 
igators, or simply not reported. 

^^Construct validation procedures may assume nomothetic as well as idiographic 
forms, the latter of which preserves intact meaning systems while ensuring 
the methodological accountability of claims about mental states. See, for 
example. Fay Fransella and Don Bannister, A Manual for Repertory Grid 
Technique (London: Academic Press, 1977), 92-104. 
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5.2 Knowledge Use as Criterion 



Issues of reproduciDility, reliability, and validity are intimately 
linked to the definition of knowledge use* What we have called the 
criterion problem^^ pervades nearly all published and unpublished literature 
on knowledge use of which we are aware. Formal definitions of knowledge 
use are typically absent; even when such definitions are provided, knowledge 
is frequently viewed simply as the equivalent of research or information, 
or as the embodiment of some normative image of scientific knowledge. 
Knowledge is often represented in terms of quasi-nliysical objects (e.g., 
research reports or information sources) or In accordance with definitions 
favored by scientists and philosophers, rather than in terms that are 
epistemically meaningful to users themselves. 

Methodological limitations are also evident in attempts to define use. 
A pervasive dualism characterizes much research in the field, a dualism 
where the uses of knowledge are divided into two classes: conceptual 



Section 2 above. 
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Such normative images of scientific knowledge, while essential for 
affirming the comparative superiority of science as a way of knowing, are 
wholly inappropriate as a basis for defining and specifying constructs that 
may be investigated through empirical research. Those who confuse descriptive 
and normative definitions of knowledge have been described as "justified true 
belief (JTB) fanatics" by Fritz Machlup in Knowledge and Knowledge Production , 
loc > cit . 

49 

A few investigators, while drawing attention to this problem of coor- 
dinating meanings of knowledge in science and everyday practice, neverthe- 
less stop short of providing workable definitions or procedures for conduct- 
ing research on "ordinary" and "scientific" knowledge. See, for example, 
David K. Cohen and Michael S. Garet, "Reforming Educational Policy With 
Applied Research," Harvard Educational Review , 45, 1(1975) :17-43; and 
Charles E. Lindblom and David K. Cohen, Usable Knowledge : Social Science 
and Social Problem Solving (New Haven: Yale University Press, 1979) . 
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and instrumental." Conceptual use is generally understood in terms of 
mental processes (e.g., problem definition), while instrumental use is 
equated with individual or collective action (e.g., the adoption of an 
innovation). Indeed, the vast majority of definitions of knowledge use 
appear to fit into one of four categories formed by the intersection of 
two dimensions: conceptual versus instrumental use and imposed versus 
generated knowledge (see Figurel).^^ 

Figure 1 
DEFINITIONS OF KNOWLEDGE USE 



Knowledge is defined in 
terms of properties that 
are: 



Imposed on Users 
(Imperative) 



Generated by Users 
(Constructed) 



The use of knowledge is defined 
in terms of processes that are: 

Conceptual Instrumental 



CONCEPTUAL 
IMPERATIVISM 



CONCEPTUAL 
CONSTRUCTIVISM 



BEHAVIORAL 
IMPERATIVISM 



BEHAVIORAL 
CONSTRUCTIVISM 
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SOURCE: Adapted from Dunn, "Usable Knowledge. 



Volume II, 



What may be called conceptual imperativism is prevalent among invest- 

52 

igators who hold particular normative images of knowledge or of more or 
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OrigiBs.j)f«tW5 d-istHiction are Nathan Caplan et. al.. The Use of Social 
Science Knowledge jn Policy Decisions at the Nationa'T'LeveiTT "T TReport to 
Respondents (Ann Arbor. MI: University of Michigan, Institute Tor Social" 
Research, Center for Utilization of Scientific Knowledge, 1975); and 
Robert F. Rich, "Selective Utilization of Social Science Related Information 
by Federal Policymakers," Inquiry , 13, 3(1975). 
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See Dunn, "Usable Knowledge...," Volume II. 
52 

Jack Knott and Aaron Wildavsky, for example, reserve the term knowledge 
for empirical certainty: "When policy makers are certain that manipulating 
these variables will product the expected effects--that is, if ''x' is done, 
^^'y' will follow with a known probability— then they have knowledge". See 
"If Dissemination is The Solution, What Is The Problem?" Knowledge . 1, 4(1980) 
515-36. 'J I I 
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less fixed psychological structures, styles, or traits. in research on 
educational innovations the technological perspective of knowledge use 
described by House comes close to conceptual imperativism, as we use this 
term, while conceptual constructivism parallels in major respects his own 
cultural perspective of knowledge use.^* Conceptual co.istructivism is also 
evident among investigators who opt for a broad, flexible, and even diffuse 
definition of knowledge use. stressing changes in perceptions, orientations, 
interpretations, and assumptions.^^ 

Behavioral imperativism. by contrast, focuses on overt actions that 
presumably are tied to relatively fixed or at least coirmon structures 
for producing, disseminating, and using knowledge. A political perspective 
of knowledge use. which includes generalizations about lawful regularities 
in bureaucratic behavior that constrain or impede knowledge use. approximates 
what is meant by behavioral imperativism.^^ By contrast, behavioral con- 
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For example, the Myers-Briggs Type Indicator has been used to characterize 
the cognitive styles or personality types of knowledge users. See Ian I 
Mitroff and Donna Mitroff. "Interpersonal Communication for Knowledge 
Utilization." Knowledge . 1. 2(1980) : 203-17. 



54 



House. "Three Perspectives on Innovation...." loc. cit. 
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and Garet, loc. cit.; David K. Cohen and Janet 
and Social PSTicy: Schools and Race." in Using 
Policy Making .' ed. Carol H. Weiss (Lexington; — 
.~d7=83; narvin C. Alkin et. al_.. Using Evaluations: 

ffei*ence? (Beverly Hills. CA: SageTu5Ti cations. 

uti nzation-Focused Evaluation (Beverly Hills. CA; 
ana taroi n. weiss. "ivieasuring the Use of Evaluation." 
Concepts and Measurement Techniques, ed. James A. 
Sage Publications, 1981). pp. 17-34. 



See. for example. Cohen 
A. Weiss. "Social Science 
Social Research in Public 
HA; u.L. H6ath. pp 
Does Evaluation Make a Di 
T97?); MIChAel g. PatTonT 
Sage Publications. 1978); 
in Utilizing Evaluation; 
Ciarlo (Beverly Hills. CA 
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For exair6le. Rich observes that "the bureaucratization of the knowledge 
inquiry system... is the critical variable for understanding levels of 
utilization and nonutil ization. . . Thus, according to this perspective, 
one would effect change in the knowledge inquiry system by influencing 
bureaucratic rules and procedures— not knowledge and/or policy/sector 
specific characteristics." Robert F. Rich, "Editor's Introduction," 
American Behavioral Scientist , 22, 3(1979)_328-29. See also House, "Three 
Perspectives...," foe, cit . 
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structivism defines knowledge use primarily in terms of overt behavior 

presumably connected to knowledge that has been generated by users. 

These four types of definitions draw attention to dualistic tendencies 

in research on knowledge use. Approximately seventy percent of the 

ferty-two studies reviewed under this projpct were based primarily on 

instrumental definitions of use, with the remainder stressing conceptual 

58 

and symbolic definitions. Instrumental ly-focused studies generally 
neglected properties related to the purposes, expected benefits, or 
underlying meanings of knowledge aad its uses. Yet even that twenty-five 
percent of studies based on a conceptual definition of use focus primarily 
on surface properties of knowledge— some eighty percent of such studies 
take the meaning of knowledge for granted. Indeed, only thirteen of 
forty-two studies attempt to elicit the subjective meanings attached 
to knowledge by users. This finding punctuates once more the relative 
paucity of reproducible qualitative procedures appropriate for studying 
knowledge use. 

5.3 Formal Classificational Schema 

A central difficulty with available definitions of knowledge use if 
their oversimpli cation of criterion dimensions. The distinction between 
conceptual and instrumental uses, together with contrasts between imposed 
and generated properties of knowledge itself, oversimplifies and thereby 
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Janet Weiss, for example, urges that we study knowledge use in terms of 
"The symbiotic relationships among policy actors, knowledge, and the 
political and institutional context of policy making." See "Access to 
Influence: Some Effects of Policy Sector on the Use of Social Science," 
American Behavioral Scientist , 22, 3(1979) :456-57. 
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See Volume III, Appendix C. 
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conceals numerous additional dimensions according to which knowledge 
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use may be classified: 



* Usership . Taking the content of knowledge as given, use may be 
classified according to the persons or groups who are users of 
knowledge. Users may b£ national or local policymakers, members of 
professional associations, media representatives, client groups, 
social scientists, and so forth. While it is difficult to imagine 
users who are not also subscribers, users often subscribe to know- 
ledge for reasons that differ from those of original subscribers. 
For example, policymakers may use research results to support 
personal or political goals, thus subscribing to knowledge for 
reasons that ehre. quite different from those of researchers. Who 
are the users of knowledge? 

* Object . Use may also be classified according to its objects, which 
may include recommendations, empirical generalizations, hypotheses, 
theories, models, concepts, assumptions, principles, ideas, and so 
forth. The object of use and the object of knowledge differ, since 
use classified by object (e.g., conceptual use) implies nothing about 
the object of knowledge. Political knowledge may be usecL^omake 
recommendations, to conceptualize a problem, to explore ideas fo' 
own intrinsic merit, and so forth* What is used? Why? y 



* Directness . Use may be classified according to its directness to an 
original knowledge source. Use may be relatively direct, in cases 
where some user reads an original report or study produced by a 

CO 

See Dunn, "Methodological Choices in Studying Knowledge Use" and Zaltman, 
"Construing Knowledge Use" in Volume II of this report. See also Weiss, 
"Measuring the Use of Evaluation," o£. cit . , pp. 24-6. 
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social scientist. Indirect use occurs when research findings are 
filtered through executive summaries and secondary written descrip- 
tions or reviews, or passed on through conversations that involve 
no face-no-face interaction ("invisible colleges"). Indirect use 
also takes place when knowledge is imbedded in developmental or 
technological production decisions. In the latter case, users 
may be unaware of the underlying evidence and assumptions that 
accompany the adoption of a technology. A lack of awareness of 
the knowledge claims/themselves makes them inaccessible for assess- 
ment and integration into meaning structures or practice. How 
close is the use of knowledge to its original source? 

* Temporal Proximity . Use may be classified according to its proximity 
in time to an original knowledge source. Apart from its directness, 
use may be immediate, occurring simultaneously with the creation of 
knowledge. Use may also be delayed, as when knowledge is transmitted 
across years, generations, or historical epochs. How inmediate is 
the use of knowledge? 

* Magnitude of Expected Effects . Use may also be classified according 
to its expected effects. Use may refer to "conceptual" and/or 
"instrumental" effects, categories which appear to be points along 

a single continuum or dimension, rather than discrete and mutually 
exclusive classes. In this context, Machlup describes thirteen elements 
of the state or act of knowing which specify, in part, what is meant 
by the magnitude of expected effects of use.^^ How much effect is 
required to count as use? 

^°Fritz Machlup, "Uses, Values, and Benefits of Knowledge," Knowledge , 1 
1(1979) :62-81., These elements include acts of knowing that range from 
being acquainted with or aware of something to being able to explain, 
demonstrate, talk about, or perform some action. While none of these 
^ elements is behavioral, being able to explain or demonstrate something is 
ERIC nevertheless instrumental to being ah.le to perform and actually take actions. 



Taking its uses as given, knowledge may be conceptualized according to 
multiple dimensions that assist in specifying the diverse meanings attached 
to the term: 

* Subscribership , Knowledge may be classified according to the persons 

or groups who subscribe to it. The dimension of subscribership includes 

such categories as "personal knowledge," "professional knowledge," 

"practitioner knowledge," "public knowledge," and so forth. In 

certain cases the use of knowledge may be virtually indistinguishable 

from subscribership, for example, when practitioners who subscribe to 

knowledge about managerial rules-of -thumb also act on that knowledge. 

Who subscribes to knowledge? Is the act of subscribing to knowledge 
identical to using it? 

* Source . Distinctions may also be based on the source of knowledge. 
Distinctions by source include "scientific knowledge," "professional 
knowledge," "craft knowledge," "practice knowledge," "experiential 
knowledge," "ordinary knowledge." Social scientists may be the source 
of knowledge while others subscribe to it, long after the same 
scientists have disavowed their original conclusions. Conversely, 
social scientists may subscribe to knowledge whose source is prac- 
tical, experimental, or ordinary. Much "scientific" knowledge 

is in part "ordinary" knowledge. From what source did knowledge 
arise? Is the source different from the subscriber? 

* Object . Knowledge may also be classified according to its object. 
Distinctions by object include "educational knowledge," "political 
knowledge," "environmental knowledge," as well as knowledge whose 
objects are particular policy issue areas or sectors--for example, 
criminal justice, housing, welfare, education, and so forth. The 
object of knowledge is independent of its subscribership and source. 
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What is the knowledge about? Are different types of objects of 
knowledge associated with use? 

* Benefit , Knowledge may also be classified in terms of the types of 
benefits expected to occur upon its use. For example, we may dis- 
tinguish types of knowledge according to expected benefits which 
are "practical," "intellectual," "pastime," "spiritual," and "un- 
wanted. "^^ The expected benefit of knowledge is not the same as its 
object, since knowledge about a particular object~e.g., the politics 
of incremental policymaking —may confer benefits that are primarily 
intellectual and only secondarily, if at all, practical. Moreover, 
the dimension of benefit is independent of subscribership and source. 
Professional knowledge (subscribership) based on experience (source) 
may be primarily intellectual or pastime knowledge (benefit) insofar 
as there is no expectation that its use will improve practice. What 
difference will the use of knowledge make? 

* Warrant . Knowledge may be classified according to the criteria or 
standards of assessment that warrant its certification as knowledge. 
Knowledge-warranting assumptions may be empirical, analytic, 
teleological, pragmatic, authoritative, ethical, and so forth. Warrants 
for knowledge ("reality tests") are organized in frames of refer- 
ence or cognitive maps that govern judgments about the adequacy, 
relevance, and cogency of knowledge. The type of warrant used 

to certify knowledge is potentially independent of its subscriber- 
ship, source, object, benefit, and purpose. "Practitioner knowledge" 
(subscribership) may be based on the same kinds of warrants employed 
by social scientists; and there is no necessary relationship between 



See Machlup, Ibid. 

ERLC 



the source of knowledge (e.g., everyday experience) and its 
theoretical adequacy or practical relevance. What makes something 
count as knowledge in the first place? 

5.4 Generative Classificational Schema 

Formal classificational schema, while helpful in enlarging the range 
of potentially available definitions, should be recognized as extensions 
of the frames of reference of researchers. Formal dimensions of know- 
ledge use, together with the constructs of which these dimensions are 
composed, are likely to be valid representations of knowledge and its 
uses only if dimensions and constructs have been coordinated with the 
meanings of users themselves. Simple questions about subscribership, 
usership, and source, as well as more complex issues surrounding expected 
effects, benefits, and warrants, are difficult or plainly impossible to 
address by studying overt behavior alone. For this reason it is 
essential to ground constructs in the knowledge-in-use of actors whose 
behavior we seek to understand. 

In pursuing this goal it would be highly useful to have generative 
classificational schema or typologies that capture. diverse meanings attached 
to knowledge and its uses. At present we are aware of no such generative 
schema, in education or in other practice areas. This methodological gap 
is not only a consequence of the relative paucity of reproducible qualitative 
procedures, but also stems from the inappropriateness of reproducible 
procedures when they are available and used. 

Thus, for example, the idea of truth tests and utility tests, while 
closely linked to the development of generative typologies, is not easily 



translated into procedures that elicit constructs which comprise individual 
and collective frames of reference. While data reduction techniques (e.g., 
factor analysis, principal components analysis, multidimensional scaling) 
may be employed to reduce large numbers of item- responses to discrete 
constructs, the fact that responses have been aggregated or averaged 
across many individuals diminishes the capacity to identify unique or dis- 
tinctively organized relations among a given person's constructs. Data 
reduction procedures also may preclude opportunities to investigate 
structural properties of frames of reference— differentiation, complexity, 
integration, permeability, orientation— all of which are central to ques- 
tions of knowledge synthesis and the dynamics of individual and collective 
learning. 

5.5 Sociocognitive Grid Procedures 

One of our major methodological findings, one which emerged gradually 

t 

over the course of the project, is that new or promising procedures are 
indeed available to develop generative typologies of knowledge use. These 
procedures, which build on the simple but powerful idea pf a grid with 
m by in constructs and elements, effectively combine sociometry and social 
network analysis with various approaches to the measurement of cognitive 

CO 

structures. These procedures, since they permit the simultaneous measure- 
ment of social and intrapersonal space, are sociocognitive in the full 



See Burkart Holzner, "Social Processes and Knowledge Synthesis," Occasional 
Paper (Pittsburgh, PA: University of Pittsburgh, Program for the Study 
of Knowledge Use, 19Q2). 
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On social network analysis see, for example, Rogers and Kincaidy loc . cit . ; 
and Dunn, Bangs, and Rahmanian, "Studying Knowledge Networks," Volume II. 
On the measurement of cognitive structures see, expecially, descriptions 
of the role repertory grid and referential grid in Volume III of this report. 
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sense of the term.^^ 

A major methodological advantage of sociocognitive grid procedures is 
that they facilitate the relational study of knowledge use. One of the 
striking characteristics of research on knowledge use undertaken to date 
is its predominantly non-relational focus. Individual users are the pre- 
dominant unit of analysis and collective properties, if they are an object 
of concern at all, are frequently inferred by aggregating data on individual 
cognitive and/or social properties that are themselves defined in terms of 
the frames of reference of researchers and research sponsors. Consequently, 
the relational and contextual nature of knowledge use is lost in various 
methodological shredders which tear respondents from their own distinctive 
contexts of intrapersonal and social space. In this context. Barton's 
metaphor is appropriate: "It is a little like a biologist putting his or 
her experimental animals through a hamburger machine and looking at 
every hundredth cell through a microscope; anatomy and physiology get lost; 
structure and function disappear and one is left with a cell biology". 



"See Patric Slater, Dimensions of Intrapersonal Space (London; Wiley, 1977), 
As Slater notes (pp. 15-27) grid procedures may be traced to early socio- 
metry (Moreno), Q-methodology (Stephenson), semantic differential (Osgood) 
and the repertory grid (Kelly). 
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This point parallels that of Rogers and Kincaid, og. cit . , p. 39, although 
our concern extends to frames of reference as organized intrapersonal spaces. 
See also Paul F. Lazarsfeld, "Sociology," in Main Trends of Research in the 
Human Sciences (Paris: UNESCO, 1970). 

^^Allen Barton, "Bringing Society Back In: Survey Research and Macro-Method- 
ology," American Behavioral Scientist , 12(1968) :l-9. Quoted in Rogers and 
Kincaid, p. 39. 
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5.6 Need for Methodological Innovation 

Research on knowledge use is a scientifically demanding, practically 
important, but relatively new field within the applied social sciences. 
Accordingly, the summary of methodological findings reported above 
should be understood in a context that recognizes the magnitude of sci- 
entific and practical challenges confronting knowledge use researchers, 
generally and in the domain of education, as well as the comparative lack 
of experience of the social and natural sciences in dealing with problems 
surrounding the creation of usable knowledge. 

This report, while it draws attention to methodological limitations 
and inadequacies of research in the field, also attempts to punctuate 
opportunities for its improvement. The three volumes of this report, taken 
as a whole, are therefore devoted primarily to the Identification of method- 
ological innovations— concepts, models, approaches, methods, techniques-- 
that promise to improve present capacities to explain, predict, and 
shape processes of knowledge use. 

A striking feature of contemporary research on knowledge use is the 
uneven quality and limited appropriateness of available research methods. 
Nevertheless, there are many new and potentially more appropriate concepts, 
models, and research procedures that have not been properly disseminated 
or further tested and refined. In this context it is essential that 
funding agencies continue and even strengthen support for methodological 
research and development on processes of knowledge use. The benefits of 
such a research policy include: 

* Increased confidence in claims about the positive and negative 
impact of applied social research, claims whose validity depends 
on the conceptual adequacy of definitions of knowledge and its 
uses by educational policymakers and practitioners. 



Enhanced capacity to specify, measure, and monitor processes 
of knowledge use, processes that involve multiple stakeholders 
with competing and frequently conflicting perspectives of problems 
for which educational innovations are advanced as solutions. 

* Increased control over the quality of research, including a 
heightened capacity to promote methodological accountability 
among investigators and those who base policy decisions on 
their findings. 



Finally, this report highlights the need for new methods that 
facilitate the development of what might be called a sociocognitive 
science of knowledge applications, a science that is centrally concerned 
with the practical consequences of scientific research and development 
for social change and individual and collective learning. Contributions 
to Volume II and Volume III of this report mark a step in this direction. 
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APPENDIX A 



ANNOTATED BIBLIOGRAPHY OF WORKING 
AND OCCASIONAL PAPERS 



I William N. Dunn and Burkart Holzner. "Knowledge Use and School Improvement: 
Conceptual Framework and Study Design" 

y' 

If ' 

The University of Pittsburgh Program for the Study of Knowledge 
Use is now conducting methodological research on knowledge use 
and school improvement under a grant from the National Institute 
of Education. The purposes of this working paper are (1) to 
identify traditions in the study of knowledge use that have 
shaped the conceptual framework and research platform of the 
Pittsburgh Program; (2) to outline the three major research 
problems that the project seeks to alleviate; (3) to specify 
the research objectives and study design that guide project 
activities; and (4) to provide a brief overview of research 
products and their significance. 



II Mary Jo Dukes. "Knowledge Use and School Improvement: A Select Annotated 
Bibliography" 

This bibliography is based on a systematic search of the liter- 
ature and contains 112 annotated citations of recent works in 
the area of knowledge utilization within the field of education. 
Each item is annotated according to one of five categories: 
(1) Provides insight into the KP&U process in education; (2) 
Identifies factors which impede KU in education; (3) Suggests 
strategies for improving KU in education; (4) Concern with 
theories of KU in education; and (5) Attempts to define the 
dependent variable "use." 



Ill Evelyn M. Fisher. "Contexts for Conducting Field Research on Knowledge 
Use and School Improvement" 

The purpose of this paper is to outline and justify an approach 
that permits investigators to sample and study key "events", 
"phases" and "stakeholders" as a means to overcoming the 
problem of identifying occasions of knowledge use for study 
or biasing the research by concentrating on a single change 
attempt. 



IV Thomas R. Mclntyre and Evelyn M. Fisher. "Qualitative Procedures for 
Research on Knowledge Use and School Improvement" 

The purpose of this paper is to critically review the 
theoretical perspectives and methodoloqical techniques of 
qualitative research. The past use and applicability of 
qualitative procedures to knowledge use and school improvement 
is explored. 
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^ "'^llltllloSks""""' ^^'^^^ ""'^ Rahmanian. "Studying Knowledge 

The purpose of this paper is to explore potential applications of 
social netwofk analysis to research on knowledge use and school 
improvement.^ The pajjer (1) outlines conceptual and methodological 
bases of social network analysis; (2) reviews some relevant past 
applications of social network analysis of the study of various 
IncfTl^ structure and behavior; and (3) proposes and 

justiries the id^a of "epistemic network analysis" as an 
appropriate and useful addition to research on social networks 
in general and to studies of knowledge use in particular. 

VI William N. Dunn and MarysJ^Dukes. "Multiattribute Procedures for 
Studying Adoption Decision Making" 

A major problem in decisions to adopt an educational innovation, 
whether as an individual or as a committee member acting on be- 
half of a larger unit, is how to assess the innovations available. 
This paper suggests that the choice of a subset of usable measures 
of assessment is a multiattribute problem, that is, a problem 
where the selection of an appropriate subset depends on the 
subjective standards of assessment applied by the stakeholders 
who attribute different properties to the innovation; compares 
and contrasts multiattribute procedures according to several 
key dimensions, including the mode, scope, ground, and focus 
of procedures for selecting assessment measures; and evaluates 
these procedures in terms both of their general appropriateness 
to complex problems of choice and their applicability to the 
specific problem of selecting a subset of maximally usable 
measures to assess the potential of the innovation. 



VII William N. Dunn. "Usable Knowledge: A Metatheory of Policy Research in 
the Social Sciences" 

In an attempt to untangle the field of knowledge use, this 
paper develops an extended typology designed to capture the most 
basic properties of present-day theories of knowledge use. 
Rather than siding with a.iy particular theory, or an ideal-type 
of which it is an illustration, each theory is taken an a 
catum". At a metatheoretical level, proportions and attendant 
corollaries are generated for making statements about ideal- 
typical theories as a whole. 



VIII Gerald Zaltman. "Construing Knowledge Use" 

This paper discusses selected constructs of the term "use" 
and the complexity that may be concealed by oversimplistic 
imagery. The way in which knowledge use is construed may 
influence attempts at measurement. Improvement in measure- 
ment techniques may result from increased differentiation in tha 
way we think about use, particularly as distinct from impact, 
as well as from more imaginative strategies for measurement. 
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William N. Dunn. "Reforms As Arguments" 



This paper contends that argument is a more appropriate analogy 
for reforms than is experiment. In an experimenting society, the 
outcomes cannot be said to be independent of the preferences 
of stakeholders in social reforms. The change in analogy directs 
attention to the knowledge transactions in which knowledge claims 
are assessed in accordance with frames of reference constituted 
by different configurations of adequacy, cogency, and relevance 
tests. 



Burkart Holzner and Leslie Salmon-Cox. "Frames of Reference and Prediction 
of Knowledge Use" 

This paper discusses the concept, frame of reference and the 
interplay between the individual's frame of reference, his or 
her location in social structure, and perceived responsibilities. 
The components of frames of references are examined for knowledge 
producers and teachers as Dotential knowledqe users. Frames of 
reference are a critical intermediary between knowledge and its use 
and therefore our attention to frames of reference is necessary 
for predicting knowledge use. 



William N. Dunn. "Methodological Choices in Studying Knowledge Use" 

This paper offers a framework of methodological choices for 
studying knowledge use, choices which go much beyond the sel- 
ection of particular methods and techniques. As such, it attempts 
to clarify the complexity in the field of knowledge use which 
has been conceptually soggy, theoretically fragmented and pro- 
cedurally poor. Methodological choices require the systematic 
consideration of alternative ways to conceptualize, define and 
explain knowledge use, as well as procedures that facilitate the 
acquisition of information required to address problems and 
hypotheses posed in accordance with prior conceptual and 
theoretical decisions. 
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AUTHOR : 

AVAILABILITY : 

. PURPOSE : 

VARIABLES : 

DESCRIPTION : 

DEVELOPMENT : 

RELIABILITY/ 
VALIDITY : 

■ ADMirriSTRATION : 

/ — 

SOURCES: 
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APPENDIX C 
STUDY PROFILE 

DIMENSION DESCRIPTION CODE 

Unit of Analysis 01 02 03 04 05 06 07 08 09 

Sampling 01 02 03 04 07 08 09 

Design 01 02 03 04 05 06 07 08 09 

Research Methods 01 02 03 04 05 06 07 08 09 

Analytic Methods 1 2 3 4 5 8 9 

Analytic Focus 12 8 9 

Reliability 12 8 

Validity 12 8 

Definition of Use 1 2 3 8 9 

Object of Use 12345689 

Practice Area and 
Population 
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